Review findings from Gemini + GPT + Plan Reviewer on PR #207 surfaced two
critical bugs plus a few design-doc deviations worth documenting.

1. Deleted-doc resurrection (Gemini, CRITICAL)

   The engine uses clean-delete: on delete, the doc's bits are cleared from
   every filter/sort bitmap AND from alive, so filter bitmaps stay clean and
   the query hot path has no `alive AND filter`. The cached prefilter bitmap
   is NOT maintained by clean-delete — it's a frozen snapshot. Between
   refreshes, deleted docs still have bits set in the cached prefilter. A
   query whose clauses match the prefilter (no extra clauses to narrow
   against clean filter bitmaps) would have acc = stale bitmap → deleted
   docs leak through.

   Fix: substitute() now takes an `&RoaringBitmap alive` argument and
   intersects the cached prefilter bitmap with alive before wrapping it in a
   BucketBitmap clause. Cost: one bitmap AND (~20-50ms at 107M) per
   substituted query — still a large win vs the 300-900ms saved by eliding
   the original clauses. Verified by:
     - unit test substitute_excludes_dead_slots_from_stale_prefilter
     - integration test deleted_docs_excluded_from_stale_prefilter (registers
       prefilter, deletes 3/10 docs without refresh, confirms substituted
       query returns 7 IDs excluding the deleted slots)

2. Tokio reactor starvation (Gemini, CRITICAL)

   handle_register_prefilter and handle_refresh_prefilter called
   engine.register_prefilter / engine.refresh_prefilter directly on the Axum
   async worker thread. compute_filters over 100M-bit bitmaps is ~300-900ms
   of CPU work — running that on a reactor thread starves the runtime and
   tanks p99 for every other request.

   Fix: wrap both in tokio::task::spawn_blocking. Errors propagate as
   JoinError → 500 Internal Server Error with message. No other behavioral
   change.

3. Or-children canonicalization (Plan Reviewer, low priority)

   Design open question #1: whether Or(A, B) and Or(B, A) should canonicalize
   identically. Current answer is "no" — we don't reorder Or children. Added
   unit test canonicalize_does_not_reorder_or_children as a regression
   backstop: if a future change enables Or sorting, the test must be
   updated deliberately.

4. Documented deviations + Phase 1 hazards in the design doc

   - Registry on ConcurrentEngine, not InnerEngine (reason: avoid snapshot
     publish on registration; clause sets are immutable so no coherence
     issue)
   - Phase 1b flag gate skipped (empty-registry early-return is the kill
     switch)
   - Cache/prefilter drift bifurcation: cache-hit path and cache-miss path
     may return slightly different results under drift; bounded by refresh
     interval; acceptable for Phase 1 per design's explicit "mild drift is
     acceptable"; Phase 2 includes prefilter version in cache key
   - Delete-drift / alive-intersection rationale

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Review findings from Gemini + GPT + Plan Reviewer on PR #207 surfaced two
critical bugs plus a few design-doc deviations worth documenting.

1. Deleted-doc resurrection (Gemini, CRITICAL)

   The engine uses clean-delete: on delete, the doc's bits are cleared from
   every filter/sort bitmap AND from alive, so filter bitmaps stay clean and
   the query hot path has no `alive AND filter`. The cached prefilter bitmap
   is NOT maintained by clean-delete — it's a frozen snapshot. Between
   refreshes, deleted docs still have bits set in the cached prefilter. A
   query whose clauses match the prefilter (no extra clauses to narrow
   against clean filter bitmaps) would have acc = stale bitmap → deleted
   docs leak through.

   Fix: substitute() now takes an `&RoaringBitmap alive` argument and
   intersects the cached prefilter bitmap with alive before wrapping it in a
   BucketBitmap clause. Cost: one bitmap AND (~20-50ms at 107M) per
   substituted query — still a large win vs the 300-900ms saved by eliding
   the original clauses. Verified by:
     - unit test substitute_excludes_dead_slots_from_stale_prefilter
     - integration test deleted_docs_excluded_from_stale_prefilter (registers
       prefilter, deletes 3/10 docs without refresh, confirms substituted
       query returns 7 IDs excluding the deleted slots)

2. Tokio reactor starvation (Gemini, CRITICAL)

   handle_register_prefilter and handle_refresh_prefilter called
   engine.register_prefilter / engine.refresh_prefilter directly on the Axum
   async worker thread. compute_filters over 100M-bit bitmaps is ~300-900ms
   of CPU work — running that on a reactor thread starves the runtime and
   tanks p99 for every other request.

   Fix: wrap both in tokio::task::spawn_blocking. Errors propagate as
   JoinError → 500 Internal Server Error with message. No other behavioral
   change.

3. Or-children canonicalization (Plan Reviewer, low priority)

   Design open question #1: whether Or(A, B) and Or(B, A) should canonicalize
   identically. Current answer is "no" — we don't reorder Or children. Added
   unit test canonicalize_does_not_reorder_or_children as a regression
   backstop: if a future change enables Or sorting, the test must be
   updated deliberately.

4. Documented deviations + Phase 1 hazards in the design doc

   - Registry on ConcurrentEngine, not InnerEngine (reason: avoid snapshot
     publish on registration; clause sets are immutable so no coherence
     issue)
   - Phase 1b flag gate skipped (empty-registry early-return is the kill
     switch)
   - Cache/prefilter drift bifurcation: cache-hit path and cache-miss path
     may return slightly different results under drift; bounded by refresh
     interval; acceptable for Phase 1 per design's explicit "mild drift is
     acceptable"; Phase 2 includes prefilter version in cache key
   - Delete-drift / alive-intersection rationale

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Wires the 5 metrics from design-prefilter-registry.md plus one extra age
gauge into the collect-on-scrape path. Atomics were already tracked on
PrefilterEntry from PR #207; this PR adds scrape-side plumbing so Grafana
can show registered count, cardinality, substitution rate, last compute
duration, refresh errors, and age-since-refresh.

- bitdex_prefilter_registered{index} — gauge, count of registered entries
- bitdex_prefilter_cardinality{index,name} — gauge, current bitmap size
- bitdex_prefilter_substitutions_total{index,name} — counter surfaced as gauge
  (matches existing cache_hits_total pattern — the value comes from an
  AtomicU64 counter on the entry, not from a live Prometheus counter)
- bitdex_prefilter_last_compute_seconds{index,name} — gauge
- bitdex_prefilter_refresh_errors_total{index,name} — gauge
- bitdex_prefilter_age_seconds{index,name} — gauge, alert candidate: if an
  SWR thread or orchestrator isn't refreshing, age will grow unbounded

Tests: lib 17/17 + integration 3/3 still green. No behavior change to the
substitute() hot path.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Wires the 5 metrics from design-prefilter-registry.md plus one extra age
gauge into the collect-on-scrape path. Atomics were already tracked on
PrefilterEntry from PR #207; this PR adds scrape-side plumbing so Grafana
can show registered count, cardinality, substitution rate, last compute
duration, refresh errors, and age-since-refresh.

- bitdex_prefilter_registered{index} — gauge, count of registered entries
- bitdex_prefilter_cardinality{index,name} — gauge, current bitmap size
- bitdex_prefilter_substitutions_total{index,name} — counter surfaced as gauge
  (matches existing cache_hits_total pattern — the value comes from an
  AtomicU64 counter on the entry, not from a live Prometheus counter)
- bitdex_prefilter_last_compute_seconds{index,name} — gauge
- bitdex_prefilter_refresh_errors_total{index,name} — gauge
- bitdex_prefilter_age_seconds{index,name} — gauge, alert candidate: if an
  SWR thread or orchestrator isn't refreshing, age will grow unbounded

Tests: lib 17/17 + integration 3/3 still green. No behavior change to the
substitute() hot path.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>